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Background / Context: 

Description of prior research and its intellectual context. 


As highlighted by the National Math Panel Report (2008), consistent results from laboratory 
studies have demonstrated that interleaving worked examples with problems to solve improves 
learning for novices. The seminal work by Sweller and Cooper (1985) reported that learners 
improved more if they spent half their time solving problems and the other half studying 
examples of how similar problems were solved. Since then, over 35 articles have been published 
which replicate and extend those findings. 

Despite the wealth of positive results, there are a number of factors pointing to the need for 
further work. For example, there is some counterevidence that it doesn’t always work or that 
certain constraints must be met. Renkl (1997) demonstrated that providing examples to students 
isn’t enough as students must also actively engage with the examples. Attention to the 
appropriate key infonnation is also important (Ross, 1989). Further, studies don’t demonstrate 
uniform improvement, students with lower prior knowledge are thought to benefit more than 
those with higher knowledge (Kayluga, Chandler, & Sweller, 2001). In addition, nearly all 
studies have been conducted in laboratory settings and those conducted in classrooms were not 
integrated into regular classroom instruction (e.g., Ward & Sweller, 1990). A critical gap in the 
literature, then, is ecologically valid studies conducted in real world classrooms. This work can 
only be carried out using classroom friendly materials that instantiate principles derived from the 
worked example literature. A similar gap occurs between research and practice on this topic - our 
review of homework assignments given by teachers in 5 school districts showed that 95% of 
items were problems to solve. Out of the 128 total items, only 6 requested explanations, and only 
1 included a worked example. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 


The present effort fills both the research and practice gap. The purpose of the work is to create 
materials and tests that can be used flexibly in classrooms and which employ worked examples 
interleaved with practice problems for students to solve. A total of 42 assignments addressing 
some pre- Algebra and 8 major Algebra topics have been developed through an iterative process. 
A combination of single unit, double-unit, and year-long studies have been undertaken. Driven 
by the interests of the participating districts as well as a deep understanding of the literature, the 
effects of AlgebraByExample on learning has been explored with particular attention to the role 
of individual differences. 

Setting and Population/Participants/Subjects: 

Description of the research location. Description of the participants in the study: who , how many, key features , or 
characteristics. 


The participating districts are all members of the Minority Student Achievement Network 
(MSAN) which is a self-formed alliance dedicated to addressing the achievement gap. Like most 
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MSAN districts, the 8 that have participated in this work are mostly located in inner-ring 
suburban areas (Ann Arbor (MI), Arlington (VA), Chapel Hill - Carrboro City (NC), Evanston 
Township (IL), Evanston / Skokie 65 (IL), Green Bay (WI), Shaker Heights (OH), the one 
exception is Madison Metropolitan School District (WI) which is urban). Although this paper 
will not cover all of the individual studies, the body of work as a whole has been conducted in 
300 classrooms with over 6000 students participating. Students are in 7 th - 10 th grades, with the 
majority in 8 th or 9 th enrolled in “regular” Algebra. The partnering districts have diverse student 
populations. In the study reported on here, 395 students are in the final sample, (204 
Experimental, 191 Control; 53% male; 41% low SES) across 28 classrooms (16 teachers). The 
ethnicity breakdown was: 33% Caucasian, 40% African American, 15% Latino, 6% Asian, and 
6% biracial. The average classroom was comprised of 64% minority students (African American, 
Latino, and biracial). 

Intervention / Program / Practice: 

Description of the intervention, program, or practice , including details of administration and duration. 

Typically, textbooks and worksheets provide students with problems to solve in order to practice 
what they’ve been taught. Like typical materials, AlgebraByExample relies on teachers to teach 
and requires that students practice on their own. However, the innovative research-based and 
reality-checked design provides additional scaffolding. AlgebraByExample assignment items 
are grouped in pairs that include an “example” item for students to study and explain and a “your 
turn” item to solve. See Figure 1 for sample items. 

Insert Figure 1 about here 

As shown, an example presents the work of a fictional student whose name signals ethnicity 
and/or gender. The presented work features either a correct or incorrect attempt indicated with a 
check or “X” mark and phrases such as “Natasha solved this problem correctly” or “Natasha 
didn’t complete the graph correctly.” 

Each example is followed by one or more questions that highlight key mathematical concepts 
students often misunderstand even after significant instructional time. Responding gives students 
practice analyzing, critiquing, and articulating mathematical arguments. Typically, students 
completed assignments in 10-20 minutes. Assignments were administered in class at the time 
deemed by the teacher to be most advantageous in relation to instruction on the relevant content 
area. 

Research Design: 

Description of the research design. 


Random assignment was conducted at the level of the individual student. Approximately half of 
the students in each participating class were randomly assigned to the example-based group, in 
which they received the example-based assignments designed for the study. The other half was 
randomly assigned to the control group, in which they received an alternate version of the 
assignments that contained the same types of problems, but no examples or self-explanation 
prompts. 
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Each student received four example-based or four control assignments during each content unit; 
each student participated in two content units for a total of 8 assignments. The content units 
across the different classrooms were: Linear Equations, Pre- Algebra, Graphing, and Quadratics. 
Students completed a pre-test and post-test consisting of identical items that included both 
conceptual and procedural problems. To assess students’ procedural knowledge, we used 4 items 
that required students to carry out procedures to solve problems; all four problems were 
isomorphic to problems in the assignments, which were representative of the types of problems 
found in Algebra I textbooks and taught in Algebra I courses. To assess students’ conceptual 
knowledge, we used 22 items that measured understanding of crucial concepts from the four 
assignments (e.g., the meaning of the equals sign, the significance of negatives in terms, 
identification of like terms, etc.; Booth & Koedinger, 2008; Kieran, 1981; Knuth et al., 2006; 
Vlassis, 2004). Sample conceptual and procedural items can be found in Figure 2. Tests and 
assignments were paper-and-pencil documents. 

Insert Figure 2 about here 


Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 


All studies were conducted in a typical course setting, with all assignments and testing done as 
part of nonnal classroom activities. Hard copies of the assignments and assessments were 
collected, coded for completion, and in the case of the assessments, for accuracy as well. Mean 
pretest and posttest scores can be found in Table 1. All analyses were conducted with 
Hierarchical Linear Modeling (HLM) software. 

To detennine whether example-based assignments improve learning for algebra students, and 
whether there are differences in benefit based on either individual minority status or class 
composition, we computed two pairs of two-level hierarchical linear models with individual 
students nested in classrooms. The first pair of models (l CO nc and l proc ) included 5 predictors: 
pretest conceptual knowledge, pretest procedural knowledge, minority status, condition, and the 
interaction between condition and minority status. The second pair of models (2 conc and 2 proc ) 
also included 5 predictors: pretest conceptual knowledge, pretest procedural knowledge, 
percentage of minority students in the classroom, condition, and the interaction between 
condition and the percentage of minority students in the classroom. Any results not reported 
failed to reach significance. 


Findings / Results: 

Description of the main findings with specific details. 


Conceptual Knowledge. In both models, posttest conceptual scores were positively 
associated with pretest conceptual scores ([3s= .25 and .27, Fs(26, 389) = 5.53 and 5.92, both ps 
< .001). In model l conc , posttest conceptual scores were inversely associated with the individual 
student’s minority status (P= - .05, F( 26, 389) = -2.25, p = .003), and in model 2 conc they were 
inversely associated with the percentage of minority students in the classroom (p= - .22, F( 26, 
389) = -3.80, p < .001). This indicates that having a higher conceptual score at pretest and either 
being a non-minority student or being in a class with a lower percentage of minority students 
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predicts higher posttest scores. No interactions between condition and either type of minority 
status infonnation were found. 

Procedural knowledge. In both models, posttest procedural scores were positively 
associated with pretest procedural scores (Ps= .23 and .24, Fs(26, 389) = 3.37 and 3.52, both ps < 
.001) and pretest conceptual scores ([3s= .24 and .27, Fs(26, 389) = 3.02 and 3.43, ps = .003 and 
.001). In model l proc , posttest procedural scores were inversely associated with the individual 
student’s minority status (P= - . 10, F( 26, 389) = -2.76, p = .007), and in model 2 proc they were 
inversely associated with the percentage of minority students in the classroom (P= - .27, F( 26, 
389) = -2.50, p = .02). This indicates that having a higher procedural score at pretest, having a 
higher conceptual score at pretest, and either being a non-minority student or being in a class 
with a lower percentage of minority students predicts higher posttest scores. In model l proc , 
there was no interaction between condition and individual minority status. However, in model 
2 proc there was a trend toward a positive association with the interaction between the slope of 
Condition and the percentage of minority students in the classroom (P= .17, F( 26, 389) = 1 .75, p 
= .08). Follow-up examination of simple slopes indicated that there was an effect of minority 
percentage in the control group (higher minority classes improved less than lower minority 
classes; t = -2.50, p = .01), but in the example-based group, students in both high and low- 
minority classes learned equally ( t = -0.68, p = .50). Put another way, there were trends toward a 
main effect of condition in higher-minority classrooms (at and above 50% minority, t=l .83, 
p=0.07). This indicates that for students in high-minority classrooms, being in the example-based 
condition predicted higher posttest scores. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 


Results demonstrate that students in high-minority populations benefit more from example-based 
assignments than do those in lower-minority groups in terms of an increase in their procedural 
perfonnance. Results also indicate that conceptual learning occurs more readily for students in 
low-minority classrooms - students in higher-minority classrooms do not tend to learn as much. 
However, for students in high-minority classrooms , being in the example-based condition leads 
to greater learning than the control condition. Thus, providing correct and incorrect worked 
examples can improve learning, perhaps especially for students who are at a disadvantage. 
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Figure 1: Excerpts from the example-based and control versions of the assignment on Solving 
Multi-Step Equations 
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Correct Example 


4 = 5 


Sasha solved this problem correctly. Here are 
the steps she used to solve the problem: 

(•*«*)' s£--u * 5 '(h»-u) -» 

3-5(H»-L) J* 


3 ; 20* -30 

*30 *» 

33 - 20 v 


Why did Sasha multiply both sides by (4x-6) in 
the highlighted step(*)? 


5. 


3x = 4x - 6 + 5 


6 . 


3 = 4 -r 6x - 5r 


Incorrect Example 

Umi tried to solve this problem, but she didn’t 
do it correctly. Here is her first step to solve the 
problem: 

3x - Mm -l« + 5 

-5 “5 

3* - 4m _ 1 1 

What was Umi’s first step to solve the problem? 


3. 


Why didn't Umi’s first step keep the equation in 
balance? 
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5. 2k = 5k -9 + 6 


6. 8 = 7 + 4v - 3v 
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Figure 2: Sample assessment items for Experiment 2 


Conceptual 


Procedural 


State whether each of the following is equivalent tox + 4 — 2 + x: 



a. 

(x + 4) - (2 + x) 

Yes 

No 

Ct 




Lx 

b. 

4+x-2+x 

Yes 

No 

a j 
OX) 

c. 

x + (4 - 2) + x 

Yes 

No 






< 

d. 

x+4-x+2 

Yes 

No 

i 

Lx 

e. 

(x + 4) + (-2 + x) 

Yes 

No 

Cm 

f. 

x+4+x-2 

Yes 

No 


g- 

x + 2(2 - 1) + x 

Yes 

No 


State whether each of the following is true for the line: (y - 3) = 2{x + 1) 


Find the quotient for the expression and write 
in simplest form. Show all of your work: 



Find the x- and y-intercepts. Then use them 
to graph the equation: 


a. 

The line goes through (1,-3) 

Yes 

No 

b. 

The slope-intercept form of the line is y = 2x + 5 

Yes 

No 

c. 

The line has a slope of 2 

Yes 

No 

d. 

The line goes through (3,-1) 

Yes 

No 

e. 

The line has a slope of X A 

Yes 

No 

f. 

The line goes through (-1,3) 

Yes 

No 

g- 

The slope-intercept form of the line is y = 2x — 1 

Yes 

No 


Identify the type of each function: 


CJ 


CS 

u 

CS 

S 

c* 


a. 

y= 

2 

-X 

+6 



Linear 

Quadratic 

Exponential 

b. 

X 

2 

4 

6 

8 

Linear 

Quadratic 

Exponential 

y 

1 

4 

7 

10 





c. 

y= 

-2 x +6 



Linear 

Quadratic 

Exponential 


X 

3 

4 

5 

6 

Linear 

Quadratic 

Exponential 


y 

11 

18 

27 

00 

m 




f. 

y = 

2x + 6 



Linear 

Quadratic 

Exponential 



Solve for z using the quadratic formula. 
Show all of your work. 

z 2 - 4z-3 
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Table 1 

Experiment 2 .-Means and Standard Deviations for Conceptual and Procedural Scores at Pretest 


and Posttest by Condition and Minority Status 



Procedural 

Conceptual 

MAP 

PAP 

PAY 

ENJ 


PC 


Pre 

Pos 

Pre 

Pos 

Pre 

Pos 

Pre 

Pos 

Pre 

Pos 

Pre 

Pos 

Pre 

Post 



t 


t 


t 


t 


t 


t 



Example 

.19 

.43 

.64 

.75 

5.1 

5.1 

4.6 

4.6 

4.5 

4.5 

4.2 

4.3 

5.1 

5.2 


(.23) 

(.26) 

(.17) 

(.16) 

(1.6) 

(1.4) 

(1.7) 

(1.6) 

(1.7) 

(1.6) 

(1.9) 

(1.7) 

(1.7) 

(1.4) 

Control 

.18 

.42 

.64 

.75 

5.0 

5.1 

4.3 

4.5 

4.2 

4.2 

4.1 

4.2 

5.0 

5.3 


(.21) 

(.29) 

(.18) 

(.17) 

(1.6) 

(1.3) 

(1.7) 

(1.5) 

(1.8) 

(1.7) 

(1.7) 

(1.6) 

(1.8) 

(1.5) 


Note. Mean(SD) 
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